Run the generate-sitemap pipeline

dpp runs the knesset data pipelines periodically on our server.

This notebook runs the generate-sitemap pipelines which generates the sitemap at https://oknesset.org/sitemap.txt

Generate the site

Run the render site pages notebook

Verify:



In [41]:

    
%%bash
echo committees
ls -lah ../data/committees/dist/dist/committees | wc -l
echo factions
ls -lah ../data/committees/dist/dist/factions | wc -l
echo meetings
ls -lah ../data/committees/dist/dist/meetings/*/* | wc -l
echo members
ls -lah ../data/committees/dist/dist/members | wc -l









    



committees
1559
factions
26
meetings
2977
members
2081

Run the generate-sitemap pipeline



In [49]:

    
!{'cd /pipelines; dpp run --verbose ./knesset/generate-sitemap'}









    



[./knesset/generate-sitemap:T_0] >>> INFO    :6d1ce4d1 RUNNING ./knesset/generate-sitemap
[./knesset/generate-sitemap:T_0] >>> INFO    :6d1ce4d1 Collecting dependencies
[./knesset/generate-sitemap:T_0] >>> INFO    :6d1ce4d1 Running async task
[./knesset/generate-sitemap:T_0] >>> INFO    :6d1ce4d1 Waiting for completion
[./knesset/generate-sitemap:T_0] >>> INFO    :6d1ce4d1 Async task starting
[./knesset/generate-sitemap:T_0] >>> INFO    :6d1ce4d1 Searching for existing caches
[./knesset/generate-sitemap:T_0] >>> INFO    :6d1ce4d1 Building process chain:
[./knesset/generate-sitemap:T_0] >>> INFO    :- generate_sitemap
[./knesset/generate-sitemap:T_0] >>> INFO    :- (sink)
[./knesset/generate-sitemap:T_0] >>> INFO    :generate_sitemap: INFO    :loading from data path: /pipelines/data/committees/dist/dist
[./knesset/generate-sitemap:T_0] >>> INFO    :generate_sitemap: INFO    :num_links_per_file=50000
[./knesset/generate-sitemap:T_0] >>> INFO    :6d1ce4d1 DONE /usr/local/lib/python3.6/site-packages/datapackage_pipelines/manager/../lib/internal/sink.py
[./knesset/generate-sitemap:T_0] >>> INFO    :6d1ce4d1 DONE /pipelines/knesset/generate_sitemap.py
[./knesset/generate-sitemap:T_0] >>> INFO    :6d1ce4d1 DONE V ./knesset/generate-sitemap {'num-directories': 43, 'num-files': 3197, 'num-sitemap-links': 3197, 'num-sitemap-txt-files': 1}
INFO    :RESULTS:
INFO    :SUCCESS: ./knesset/generate-sitemap {'num-directories': 43, 'num-files': 3197, 'num-sitemap-links': 3197, 'num-sitemap-txt-files': 1}

View the sitemap



In [43]:

    
%%bash
echo number of committees: `cat ../data/committees/dist/dist/sitemap.txt | grep committees | wc -l`
echo first 10 committees:
cat ../data/committees/dist/dist/sitemap.txt | grep committees | head
echo number of meetings: `cat ../data/committees/dist/dist/sitemap.txt | grep meetings | wc -l`
echo first 10 meetings:
cat ../data/committees/dist/dist/sitemap.txt | grep meetings | head









    



number of committees: 778
first 10 committees:
https://oknesset.org/committees/965.html
https://oknesset.org/committees/928.html
https://oknesset.org/committees/index.html
https://oknesset.org/committees/109.html
https://oknesset.org/committees/430.html
https://oknesset.org/committees/1004.html
https://oknesset.org/committees/23.html
https://oknesset.org/committees/126.html
https://oknesset.org/committees/123.html
https://oknesset.org/committees/711.html
number of meetings: 1002
first 10 meetings:
https://oknesset.org/meetings/4/2/425865.html
https://oknesset.org/meetings/4/2/428527.html
https://oknesset.org/meetings/4/2/422217.html
https://oknesset.org/meetings/4/2/425287.html
https://oknesset.org/meetings/4/2/429615.html
https://oknesset.org/meetings/4/2/425155.html
https://oknesset.org/meetings/4/2/426910.html
https://oknesset.org/meetings/4/2/425961.html
https://oknesset.org/meetings/4/2/424526.html
https://oknesset.org/meetings/4/2/426405.html